Systems and methods for processing electronic images for biomarker localization

ABSTRACT

Systems and methods are disclosed for receiving digital images of a pathology specimen from a patient, the pathology specimen comprising tumor tissue, the one or more digital images being associated with data about a plurality of biomarkers in the tumor tissue and data about a surrounding invasive margin around the tumor tissue; identifying the tumor tissue and the surrounding invasive margin region to be analyzed for each of the one or more digital images; generating, using a machine learning model on the one or more digital images, at least one inference of a presence of the plurality of biomarkers in the tumor tissue and the surrounding invasive margin region; determining a spatial relationship of each of the plurality of biomarkers identified in the tumor tissue and the surrounding invasive margin region to themselves and to other cell types; and determining a prediction for a treatment outcome and/or at least one treatment recommendation for the patient.

RELATED APPLICATION(S)

This application is a continuation of U.S. Non-provisional applicationSer. No. 17/519,106 filed Nov. 4, 2021, which is a continuation of U.S.Non-provisional application Ser. No. 17/160,127 filed Jan. 27, 2021,which claims priority to U.S. Provisional Application No. 62/966,723filed Jan. 28, 2020, the entire disclosures of which are herebyincorporated herein by reference in their entireties.

FIELD OF THE DISCLOSURE

Various embodiments of the present disclosure pertain generally tolocalization of biomarkers and/or inferring spatial relationships in adigital pathology slide. More specifically, particular embodiments ofthe present disclosure relate to systems and methods for tumor andinvasive margin detection, localized biomarker prediction, and/orbiomarker and spatial relationship comparison. The present disclosurefurther provides systems and methods for using artificial intelligence(AI) to spatially infer various genomic features, molecular tests, andother analyses.

BACKGROUND

Comprehensive genetic and molecular testing of cancer tissue may allowfor precision treatment of solid tumors via targeted therapies. Eventhough the cost of genome sequencing has substantially decreased overthe years, these tests are still costly, slow, and require substantialamount of tissue that is quite limited in clinical studies. Hematoxylinand Eosin (H&E) staining is affordable and provides a comprehensivevisual description of the tumor and its microenvironment.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of thedisclosure. The background description provided herein is for thepurpose of generally presenting the context of the disclosure. Unlessotherwise indicated herein, the materials described in this section arenot prior art to the claims in this application and are not admitted tobe prior art, or suggestions of the prior art, by inclusion in thissection.

SUMMARY

According to certain aspects of the present disclosure, systems andmethods are disclosed for biomarker localization within tumormicroenvironment and in the invasive margin of the tumor usingartificial intelligence (AI).

A computer-implemented method for analyzing an image corresponding to aspecimen, includes: receiving one or more digital images of a pathologyspecimen from a patient, the pathology specimen comprising tumor tissue,the one or more digital images being associated with data about aplurality of biomarkers in the tumor tissue and data about a surroundinginvasive margin around the tumor tissue; identifying the tumor tissueand the surrounding invasive margin region to be analyzed for each ofthe one or more digital images; generating, using a machine learningmodel on the one or more digital images, at least one inference of apresence of the plurality of biomarkers in the tumor tissue and thesurrounding invasive margin region; determining a spatial relationshipof each of the plurality of biomarkers identified in the tumor tissueand the surrounding invasive margin region to themselves and other celltypes; and determining, based on the spatial relationship of each of theplurality of biomarkers to themselves or other cell types, a predictionfor a treatment outcome and/or at least one treatment recommendation forthe patient.

In accordance with another embodiment, a system for analyzing an imagecorresponding to a specimen, includes: receiving one or more digitalimages of a pathology specimen from a patient, the pathology specimencomprising tumor tissue, the one or more digital images being associatedwith data about a plurality of biomarkers in the tumor tissue and dataabout a surrounding invasive margin around the tumor tissue; identifyingthe tumor tissue and the surrounding invasive margin region to beanalyzed for each of the one or more digital images; generating, using amachine learning model on the one or more digital images, at least oneinference of a presence of the plurality of biomarkers in the tumortissue and the surrounding invasive margin region; determining a spatialrelationship of each of the plurality of biomarkers identified in thetumor tissue and the surrounding invasive margin region to themselvesand other cell types; and determining, based on the spatial relationshipof each of the plurality of biomarkers to themselves and other celltypes, a prediction for a treatment outcome and/or at least onetreatment recommendation for the patient.

In accordance with another embodiment, at least one non-transitorycomputer-readable medium storing instructions performing a method foranalyzing an image corresponding to a specimen, the at least onenon-transitory computer readable medium storing instructions which, whenexecuted by one or more processors, cause the one or more processors toperform operations including: receiving one or more digital images of apathology specimen from a patient, the pathology specimen comprisingtumor tissue, the one or more digital images being associated with dataabout a plurality of biomarkers in the tumor tissue and data about asurrounding invasive margin around the tumor tissue; identifying thetumor tissue and the surrounding invasive margin region to be analyzedfor each of the one or more digital images; generating, using a machinelearning model on the one or more digital images, at least one inferenceof a presence of the plurality of biomarkers in the tumor tissue and thesurrounding invasive margin region; determining a spatial relationshipof each of the plurality of biomarkers identified in the tumor tissueand the surrounding invasive margin region to themselves and other celltypes; and determining, based on the spatial relationship of each of theplurality of biomarkers to themselves and other cell types, a predictionfor a treatment outcome and/or at least one treatment recommendation forthe patient.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1A illustrates an exemplary block diagram of a system and networkfor localizing biomarkers and inferring spatial relationships, usingmachine learning, according to an exemplary embodiment of the presentdisclosure.

FIG. 1B illustrates an exemplary block diagram of a biomarkerlocalization platform, according to techniques presented herein.

FIG. 1C illustrates an exemplary block diagram of a slide analysis tool,according to techniques presented herein.

FIG. 2A is a flowchart illustrating an exemplary method for use of abiomarker localization within tumor microenvironment using AI, accordingto techniques presented herein.

FIG. 2B is a flowchart illustrating an exemplary method for training andusing a tumor and invasive margin detection module using AI, accordingto techniques presented herein.

FIG. 2C is a flowchart illustrating an exemplary method for training andusing a localized biomarker prediction module, according to techniquespresented herein.

FIG. 2D is a flowchart illustrating an exemplary method for training andusing a biomarker comparison module, according to techniques presentedherein.

FIG. 3 is an exemplary system trained to detect immune markers within atumor and surrounding invasive margin, according to techniques presentedherein.

FIG. 4 is a flowchart illustrating an exemplary method for training andusing a machine learning model to characterize immune markers in H&E,according to techniques presented herein.

FIG. 5 is a flowchart illustrating an exemplary method for training andusing a machine learning model for localization of gene signaturesand/or mutations in pathology specimens, according to techniquespresented herein.

FIG. 6 is a flowchart illustrating an exemplary method for training andusing a machine learning model for localization of biomarkers in a tumormicroenvironment for immunotherapy response prediction, according totechniques presented herein.

FIG. 7 is a flowchart illustrating an exemplary method for training andusing a machine learning model to predict antineoplastic resistance,according to techniques presented herein.

FIG. 8 depicts an example system that may execute techniques presentedherein.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of thepresent disclosure, examples of which are illustrated in theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.

The systems, devices, and methods disclosed herein are described indetail by way of examples and with reference to the figures. Theexamples discussed herein are examples only and are provided to assistin the explanation of the apparatuses, devices, systems, and methodsdescribed herein. None of the features or components shown in thedrawings or discussed below should be taken as mandatory for anyspecific implementation of any of these devices, systems, or methodsunless specifically designated as mandatory.

Also, for any methods described, regardless of whether the method isdescribed in conjunction with a flow diagram, it should be understoodthat unless otherwise specified or required by context, any explicit orimplicit ordering of steps performed in the execution of a method doesnot imply that those steps must be performed in the order presented butinstead may be performed in a different order or in parallel.

As used herein, the term “exemplary” is used in the sense of “example,”rather than “ideal.” Moreover, the terms “a” and “an” herein do notdenote a limitation of quantity, but rather denote the presence of oneor more of the referenced items.

Pathology refers to the study of diseases. More specifically, pathologyrefers to performing tests and analysis that are used to diagnosediseases. For example, tissue samples may be placed onto slides to beviewed under a microscope by a pathologist (e.g., a physician that is anexpert at analyzing tissue samples to determine whether anyabnormalities exist). That is, pathology specimens may be cut intomultiple sections, stained, and prepared as slides for a pathologist toexamine and render a diagnosis. When uncertain of a diagnostic findingon a slide, a pathologist may order additional cut levels, stains, orother tests to gather more information from the tissue. Technician(s)may then create new slide(s) that may contain the additional informationfor the pathologist to use in making a diagnosis. This process ofcreating additional slides may be time-consuming, not only because itmay involve retrieving the block of tissue, cutting it to make a new aslide, and then staining the slide, but also because it may be batchedfor multiple orders. This may significantly delay the final diagnosisthat the pathologist renders. In addition, even after the delay, theremay still be no assurance that the new slide(s) will have informationsufficient to render a diagnosis.

Pathologists may evaluate cancer and other disease pathology slides inisolation. The present disclosure presents a consolidated workflow forimproving diagnosis of cancer and other diseases. The workflow mayintegrate, for example, slide evaluation, tasks, image analysis andcancer detection artificial intelligence (AI), annotations,consultations, and recommendations in one workstation. In particular,the present disclosure describes various exemplary user interfacesavailable in the workflow, as well as AI tools that may be integratedinto the workflow to expedite and improve a pathologist's work.

For example, computers may be used to analyze an image of a tissuesample to quickly identify whether additional information may be neededabout a particular tissue sample, and/or to highlight to a pathologistan area in which he or she should look more closely. Thus, the processof obtaining additional stained slides and tests may be doneautomatically before being reviewed by a pathologist. When paired withautomatic slide segmenting and staining machines, this may provide afully automated slide preparation pipeline. This automation may have, atleast, the benefits of (1) minimizing an amount of time wasted by apathologist determining a slide to be insufficient to make a diagnosis,(2) minimizing the (average total) time from specimen acquisition todiagnosis by avoiding the additional time between when additional testsare ordered and when they are produced, (3) reducing the amount of timeper recut and the amount of material wasted by allowing recuts to bedone while tissue blocks (e.g., pathology specimens) are in a cuttingdesk, (4) reducing the amount of tissue material wasted/discarded duringslide preparation, (5) reducing the cost of slide preparation bypartially or fully automating the procedure, (6) allowing automaticcustomized cutting and staining of slides that would result in morerepresentative/informative slides from samples, (7) allowing highervolumes of slides to be generated per tissue block, contributing to moreinformed/precise diagnoses by reducing the overhead of requestingadditional testing for a pathologist, and/or (8) identifying orverifying correct properties (e.g., pertaining to a specimen type) of adigital pathology image, etc.

The process of using computers to assist pathologists is known ascomputational pathology. Computing methods used for computationalpathology may include, but are not limited to, statistical analysis,autonomous or machine learning, and AI. AI may include, but is notlimited to, deep learning, neural networks, classifications, clustering,and regression algorithms. By using computational pathology, lives maybe saved by helping pathologists improve their diagnostic accuracy,reliability, efficiency, and accessibility. For example, computationalpathology may be used to assist with detecting slides suspicious forcancer, thereby allowing pathologists to check and confirm their initialassessments before rendering a final diagnosis.

Histopathology refers to the study of a specimen that has been placedonto a slide. For example, a digital pathology image may be comprised ofa digitized image of a microscope slide containing the specimen (e.g., asmear). One method a pathologist may use to analyze an image on a slideis to identify nuclei and classify whether a nucleus is normal (e.g.,benign) or abnormal (e.g., malignant). To assist pathologists inidentifying and classifying nuclei, histological stains may be used tomake cells visible. Many dye-based staining systems have been developed,including periodic acid-Schiff reaction, Masson's trichrome, nissl andmethylene blue, and Hematoxylin and Eosin (H&E). For medical diagnosis,H&E is a widely used dye-based method, with hematoxylin staining cellnuclei blue, eosin staining cytoplasm and extracellular matrix pink, andother tissue regions taking on variations of these colors. In manycases, however, H&E-stained histologic preparations do not providesufficient information for a pathologist to visually identify biomarkersthat can aid diagnosis or guide treatment. In this situation, techniquessuch as immunohistochemistry (IHC), immunofluorescence, in situhybridization (ISH), or fluorescence in situ hybridization (FISH), maybe used. IHC and immunofluorescence involve, for example, usingantibodies that bind to specific antigens in tissues enabling the visualdetection of cells expressing specific proteins of interest, which canreveal biomarkers that are not reliably identifiable to trainedpathologists based on the analysis of H&E stained slides. ISH and FISHmay be employed to assess the number of copies of genes or the abundanceof specific RNA molecules, depending on the type of probes employed(e.g. DNA probes for gene copy number and RNA probes for the assessmentof RNA expression). If these methods also fail to provide sufficientinformation to detect some biomarkers, genetic testing of the tissue maybe used to confirm if a biomarker is present (e.g., overexpression of aspecific protein or gene product in a tumor, amplification of a givengene in a cancer).

A digitized image may be prepared to show a stained microscope slide,which may allow a pathologist to manually view the image on a slide andestimate a number of stained abnormal cells in the image. However, thisprocess may be time consuming and may lead to errors in identifyingabnormalities because some abnormalities are difficult to detect.Computational processes and devices may be used to assist pathologistsin detecting abnormalities that may otherwise be difficult to detect.For example, AI may be used to predict biomarkers (such as theover-expression of a protein and/or gene product, amplification, ormutations of specific genes) from salient regions within digital imagesof tissues stained using H&E and other dye-based methods. The images ofthe tissues could be whole slide images (WSI), images of tissue coreswithin microarrays or selected areas of interest within a tissuesection. Using staining methods like H&E, these biomarkers may bedifficult for humans to visually detect or quantify without the aid ofadditional testing. Using AI to infer these biomarkers from digitalimages of tissues has the potential to improve patient care, while alsobeing faster and less expensive.

The detected biomarkers or the image alone could then be used torecommend specific cancer drugs or drug combination therapies to be usedto treat a patient. The AI may identify which drugs or drug combinationsare unlikely to be successful by correlating the detected biomarkerswith a database of treatment options. This can be used to facilitate theautomatic recommendation of immunotherapy or targeted treatments for apatient's specific cancer. Further, this could be used for enablingpersonalized cancer treatment for specific subsets of patients and/orrarer cancer types.

As described above, computational pathology processes and devices of thepresent disclosure may provide an integrated platform allowing a fullyautomated process including data ingestion, processing and viewing ofdigital pathology images via a web-browser or other user interface,while integrating with a laboratory information system (LIS). Further,clinical information may be aggregated using cloud-based data analysisof patient data. The data may come from hospitals, clinics, fieldresearchers, etc., and may be analyzed by machine learning, computervision, natural language processing, and/or statistical algorithms to doreal-time monitoring and forecasting of health patterns at multiplegeographic specificity levels.

The digital pathology images described above may be stored with tagsand/or labels pertaining to the properties of the specimen or image ofthe digital pathology image, and such tags/labels may be incorrect orincomplete. Accordingly, the present disclosure is directed to systemsand methods for identifying or verifying correct properties (e.g.,pertaining to a specimen type) of a digital pathology image. Inparticular, the disclosed systems and methods may automatically predictthe specimen or image properties of a digital pathology image, withoutrelying on the stored tags/labels. Further, the present disclosure isdirected to systems and methods for quickly and correctly identifyingand/or verifying a specimen type of a digital pathology image, or anyinformation related to a digital pathology image, without necessarilyaccessing an LIS or analogous information database. One embodiment ofthe present disclosure may include a system trained to identify variousproperties of a digital pathology image, based on datasets of priordigital pathology images. The trained system may provide aclassification for a specimen shown in a digital pathology image. Theclassification may help to provide treatment or diagnosis prediction(s)for a patient associated with the specimen.

This disclosure includes one or more embodiments of a specimenclassification tool. The input to the tool may include a digitalpathology image and any relevant additional inputs. Outputs of the toolmay include global and/or local information about the specimen. Aspecimen may include a biopsy or surgical resection specimen.

Exemplary global outputs of the disclosed tool(s) may containinformation about an entire image, e.g., the specimen type, the overallquality of the cut of the specimen, the overall quality of the glasspathology slide itself, and/or tissue morphology characteristics.Exemplary local outputs may indicate information in specific regions ofan image, e.g., a particular image region may be classified as havingblur or a crack in the slide. The present disclosure includesembodiments for both developing and using the disclosed specimenclassification tool(s), as described in further detail below.

The present disclosure uses artificial intelligence (AI) to inferspatially localized genetic, molecular (e.g., the over-expression of aprotein and/or a gene product, amplification, mutations of specificgenes), flow cytometry and immune markers (tumor infiltratinglymphocytes, macrophages, etc.) from digital images of stained pathologyspecimens. The images of the tissues could be whole slide images (WSI),images of tissue cores within microarrays or selected areas of interestwithin a tissue section. Localization of biomarkers from digital imagesof tissues may have the potential to develop faster, cheaper as well asnewer/more novel diagnostic tests. Furthermore, localization ofbiomarkers from both tumor tissue and surrounding tumor tissue (invasivemargin) may have prognostic value. For example, the amount of tumorinfiltrating lymphocytes (TILs) within and in the invasive margin of atumor has prognostic value, and may be used to determine which patientswill be likely to respond to immunotherapies (e.g., Immunoscore).Understanding spatial relationships of one or more biomarkers within atumor and the invasive margin of the tumor to themselves and other celltypes may enable better treatments and more accurate patientstratification strategies.

The present embodiments may use AI to spatially infer various genomic,molecular tests from stained histologic sections, thus allowingmultiplex analysis. After localizing the biomarkers, spatialrelationships of these biomarkers to themselves and to other cell typesmay be investigated. The spatial relationships may be predictive ofcancer outcomes and therapies. Furthermore, a comprehensive analysisthat involves localizing tumor markers within a surrounding area(invasive margin) of the tumor may facilitate better understanding oftumor biology and enable development of new and novel biomarkers andtreatments.

The present embodiments may provide tumor region and invasive margindetection that may be used to determine the spatial location ofbiomarkers of diagnostic relevance. A genetic or molecular test obtainedfrom a cancer tissue may utilize the tumor region and invasive margindetection embodiments to confine the analysis to a relevant region.

FIG. 1A illustrates a block diagram of a system and network forlocalizing biomarkers and inferring spatial relationships, using machinelearning, according to an exemplary embodiment of the presentdisclosure.

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and/or laboratory information systems 125may create or otherwise obtain images of one or more patients' cytologyspecimen(s), histopathology specimen(s), slide(s) of the cytologyspecimen(s), digitized images of the slide(s) of the histopathologyspecimen(s), or any combination thereof. The physician servers 121,hospital servers 122, clinical trial servers 123, research lab servers124, and/or laboratory information systems 125 may also obtain anycombination of patient-specific information, such as age, medicalhistory, cancer treatment history, family history, past biopsy orcytology information, etc. The physician servers 121, hospital servers122, clinical trial servers 123, research lab servers 124, and/orlaboratory information systems 125 may transmit digitized slide imagesand/or patient-specific information to server systems 110 over theelectronic network 120. Server system(s) 110 may include one or morestorage devices 109 for storing images and data received from at leastone of the physician servers 121, hospital servers 122, clinical trialservers 123, research lab servers 124, and/or laboratory informationsystems 125. Server systems 110 may also include processing devices forprocessing images and data stored in the storage devices 109. Serversystems 110 may further include one or more machine learning tool(s) orcapabilities. For example, the processing devices may include a machinelearning tool for a biomarker localization platform 100, according toone embodiment. Alternatively or in addition, the present disclosure (orportions of the system and methods of the present disclosure) may beperformed on a local processing device (e.g., a laptop).

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and/or laboratory information systems 125refer to systems used by pathologists for reviewing the images of theslides. In hospital settings, tissue type information may be stored in aLIS 125. However, the correct tissue classification information is notalways paired with the image content. Additionally, even if an LIS isused to access the specimen type for a digital pathology image, thislabel may be incorrect due to the fact that many components of an LISmay be manually inputted, leaving a large margin for error. According toan exemplary embodiment of the present disclosure, a specimen type maybe identified without needing to access the LIS 125, or may beidentified to possibly correct LIS 125. For example, a third party maybe given anonymized access to the image content without thecorresponding specimen type label stored in the LIS. Additionally,access to LIS content may be limited due to its sensitive content.

FIG. 1B illustrates an exemplary block diagram of a biomarkerlocalization platform 100 for determining specimen property or imageproperty information pertaining to digital pathology image(s), usingmachine learning.

Specifically, FIG. 1B depicts components of the biomarker localizationplatform 100, according to one embodiment. For example, the biomarkerlocalization platform 100 may include a slide analysis tool 101, a dataingestion tool 102, a slide intake tool 103, a slide scanner 104, aslide manager 105, a storage 106, and a viewing application tool 108.

The slide analysis tool 101, as described below, refers to a process andsystem for determining specimen property or image property informationpertaining to digital pathology image(s), and using machine learning toclassify a specimen, according to an exemplary embodiment.

The data ingestion tool 102 refers to a process and system forfacilitating a transfer of the digital pathology images to the varioustools, modules, components, and devices that are used for classifyingand processing the digital pathology images, according to an exemplaryembodiment.

The slide intake tool 103 refers to a process and system for scanningpathology images and converting them into a digital form, according toan exemplary embodiment. The slides may be scanned with slide scanner104, and the slide manager 105 may process the images on the slides intodigitized pathology images and store the digitized images in storage106.

The viewing application tool 108 refers to a process and system forproviding a user (e.g., pathologist) with specimen property or imageproperty information pertaining to digital pathology image(s), accordingto an exemplary embodiment. The information may be provided throughvarious output interfaces (e.g., a screen, a monitor, a storage device,and/or a web browser, etc.).

The slide analysis tool 101, and each of its components, may transmitand/or receive digitized slide images and/or patient information toserver systems 110, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125 over a network 120. Further, server systems 110may include storage devices for storing images and data received from atleast one of the slide analysis tool 101, the data ingestion tool 102,the slide intake tool 103, the slide scanner 104, the slide manager 105,and viewing application tool 108. Server systems 110 may also includeprocessing devices for processing images and data stored in the storagedevices. Server systems 110 may further include one or more machinelearning tool(s) or capabilities, e.g., due to the processing devices.Alternatively or in addition, the present disclosure (or portions of thesystem and methods of the present disclosure) may be performed on alocal processing device (e.g., a laptop).

Any of the above devices, tools, and modules may be located on a devicethat may be connected to an electronic network 120, such as the Internetor a cloud service provider, through one or more computers, servers,and/or handheld mobile devices.

FIG. 1C illustrates an exemplary block diagram of a slide analysis tool101, according to an exemplary embodiment of the present disclosure. Theslide analysis tool 101 may include a training image platform 131 and/ora target image platform 135.

According to one embodiment, the training image platform 131 may includea training image intake module 132, a quality score determiner module133, and/or a treatment identification module 134.

The training image platform 131, according to one embodiment, may createor receive training images that are used to train a machine learningmodel to effectively analyze and classify digital pathology images. Forexample, the training images may be received from any one or anycombination of the server systems 110, physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125. Images used for training maycome from real sources (e.g., humans, animals, etc.) or may come fromsynthetic sources (e.g., graphics rendering engines, 3D models, etc.).Examples of digital pathology images may include (a) digitized slidesstained with a variety of stains, such as (but not limited to) H&E,Hematoxylin alone, IHC, molecular pathology, etc.; and/or (b) digitizedtissue samples from a 3D imaging device, such as microCT.

The training image intake module 132 may create or receive a datasetcomprising one or more training images corresponding to either or bothof images of a human tissue and images that are graphically rendered.For example, the training images may be received from any one or anycombination of the server systems 110, physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125. This dataset may be kept on adigital storage device. The quality score determiner module 133 mayidentify quality control (QC) issues (e.g., imperfections) for thetraining images at a global or local level that may greatly affect theusability of a digital pathology image. For example, the quality scoredeterminer module may use information about an entire image, e.g., thespecimen type, the overall quality of the cut of the specimen, theoverall quality of the glass pathology slide itself, or tissuemorphology characteristics, and determine an overall quality score forthe image. The treatment identification module 134 may analyze images oftissues and determine which digital pathology images have treatmenteffects (e.g., post-treatment) and which images do not have treatmenteffects (e.g., pre-treatment). It is useful to identify whether adigital pathology image has treatment effects because prior treatmenteffects in tissue may affect the morphology of the tissue itself. MostLIS do not explicitly keep track of this characteristic, and thusclassifying specimen types with prior treatment effects can be desired.

According to one embodiment, the target image platform 135 may include atarget image intake module 136, a specimen detection module 137, and anoutput interface 138. The target image platform 135 may receive a targetimage and apply the machine learning model to the received target imageto determine a characteristic of a target specimen. For example, thetarget image may be received from any one or any combination of theserver systems 110, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125. The target image intake module 136 may receivea target image corresponding to a target specimen. The specimendetection module 137 may apply the machine learning model to the targetimage to determine a characteristic of the target specimen. For example,the specimen detection module 137 may detect a specimen type of thetarget specimen. The specimen detection module 137 may also apply themachine learning model to the target image to determine a quality scorefor the target image. Further, the specimen detection module 137 mayapply the machine learning model to the target specimen to determinewhether the target specimen is pre-treatment or post-treatment.

The output interface 138 may be used to output information about thetarget image and the target specimen. (e.g., to a screen, monitor,storage device, web browser, etc.).

FIG. 2A is a flowchart illustrating an exemplary method for use of abiomarker localization within tumor microenvironments using AI,according to an exemplary embodiment of the present disclosure. Forexample, an exemplary method 20 (e.g., steps 21-31) may be performed byslide analysis tool 101 automatically or in response to a request from auser.

According to one embodiment, the exemplary method 20 for localizing abiomarker and inferring relationships may include one or more of thefollowing steps. In step 21, the method may include receiving one ormore digital images associated with a pathology specimen, wherein thepathology specimen comprises information about a biomarker in a tumortissue and a surrounding invasive margin associated with the one or moredigital images. The pathology specimen may comprise a histologyspecimen, a cytology specimen, etc. The one or more digital images maybe received into a digital storage device (e.g., hard drive, networkdrive, cloud storage, RAM, etc.). To train a machine learning model,each image may be paired with information about the biomarkers in thetumor and surrounding invasive margin tissues associated with eachrespective image. The information may be identified from genetictesting, flow cytometry, IHC, etc. analyzed by a pathologist,pathologist measurements, etc. A machine learning model may comprise amachine learning algorithm.

In step 23, the method may include identifying the tumor tissue and thesurrounding invasive margin region to be analyzed for each of the one ormore digital images. This may be done manually by a human orautomatically using AI.

In step 25, the method may include generating at least one inference ofa biomarker presence using a machine learning model. The method may alsoinclude using computer vision. The biomarker may be present in the tumortissue and the surrounding invasive margin image region(s). A predictionfrom the at least one inference may be output to an electronic storagedevice. An embodiment may involve generating an alert to notify a userof the presence or absence of one or more of the biomarkers.

In step 27, the method may include comparing at least one biomarker anda spatial relationship (e.g., a relative position or proximity ofclusters within biomarkers and/or to other cell types, etc.) identifiedin the tumor and the surrounding invasive margin region. Various studieshave demonstrated metrics based on spatial relationship of variousbiomarkers and/or to other cell types within the tumor and thesurrounding invasive margin can provide insights on cancer recurrence,metastasis and treatment response.

In step 29, the method may include determining a prediction for atreatment outcome and at least one treatment recommendation.

In step 31, the method may include, upon determining the prediction,generating an alert to a user. The alert may be a visual popup, a noise,or any other suitable alert method.

FIG. 2B is a flowchart illustrating an exemplary method for training andusing a tumor and invasive margin detection module using machinelearning, according to an exemplary embodiment of the presentdisclosure. The prevalence of biomarkers within a tumor is of interestand tumor regions may take up only a small fraction of the entire image.With new advances in immunotherapies, it has been demonstrated thatcellular activity in the invasive region (e.g., neighboring non-tumorregions of the tumor) may also provide valuable information on outcome.Hence, it may be crucial to identify biomarkers both in a tumor and inits invasive margin (neighboring non tumor regions). These regions ofinterest may be specified by a human expert using an image segmentation,a bounding box, or a polygon. Alternatively, a complete end-to-endsolution may include using AI to identify the appropriate locations ofsuch regions of interest. Automatic tumor and invasive marginidentification may enable a downstream AI system to learn how to detectbiomarkers from less annotated data and to make more accuratepredictions.

There may be two general approaches to creating a tumor and invasivemargin detector: strongly supervised methods that may identify preciselywhere the biomarkers could be found and weakly supervised methods thatmay not provide a precise location. During training, the stronglysupervised system may receive, as input, an image. The stronglysupervised system may further receive, for the image, a location of atumor and invasive margin region(s) that expresses the biomarker. Theselocations may be specified with pixel-level labeling, tile-levellabeling, bounding box-based labeling, polygon-based labeling, or usinga corresponding image where the tumor and the invasive margins have beenidentified (e.g., using Immunohistochemistry (IHC)). The weaklysupervised system may receive, as input, an image and thepresence/absence of a tumor and invasive regions in the image. The exactlocation of the tumor and the invasive margin locations may not bespecified in the input for the weakly supervised system. The weaklysupervised system can then be run in a localized way over regions todetermine tumor, invasive margin, and non-tumor regions. For neuralnetwork and end-to-end learning approaches, evidence visualizationmethods (e.g., GradCAM) can be utilized to localize tumor, invasivemargin and non-tumor tissue regions.

According to one embodiment, the exemplary methods 200 and 210 fortraining and using the tumor and invasive margin detection module mayinclude one or more of the following steps. In step 202, the method mayinclude receiving one or more training digital images associated with atraining pathology specimen and an associated indication of a presenceor an absence of a tumor region. The training pathology specimen maycomprise a histology specimen, a cytology specimen, etc. The trainingdigital images may be received into a digital storage device (e.g., harddrive, network drive, cloud storage, RAM, etc.).

In step 204, the method may include breaking the one or more trainingdigital images into at least one sub-region to determine if the tumor ispresent in the at least one sub-region. Sub-regions may be specified ina variety of methods, including creating tiles of the image,segmentation-based on edges or contrast, segmentations via colordifferences, supervised determination by the machine learning model,EdgeBoxes, etc.

In step 206, the method may include training the machine learning modelthat takes, as input, one of the one or more training digital imagesassociated with the pathology specimen and predicts whether the tumor ispresent. A number of methods may be used to learn which image regionsshow tumor tissue and which regions show invasive margin(s), includingbut not limited to:

Weak supervision: training a machine learning model (e.g., multi-layerperceptron (MLP), convolutional neural network (CNN), graph neuralnetwork, support vector machine (SVM), random forest, etc.) usingmultiple instance learning (MIL) using weak labeling of the digitalimage or a collection of images. The label may correspond to thepresence or absence of a tumor region.

Bounding box or polygon-based supervision: training a machine learningmodel (e.g., R-CNN, Faster R-CNN, Selective Search) using bounding boxesor polygons that specify the sub-regions of the digital image that showtumor tissue or invasive margins.

Pixel-level labeling (e.g., a semantic or instance segmentation):training a machine learning model (e.g., Mask R-CNN, U-net, FullyConvolutional Neural Network) using a pixel-level labeling, whereindividual pixels may be identified as showing tumor tissue or invasivemargins.

Using a corresponding, but different digital image that identifies tumortissue regions: a digital image of tissue that highlights the tumorregion and invasive margin (e.g., tumor/invasive margin identified usingIHC) may be registered with the input digital image. For example, adigital image of an H&E image could be registered or aligned with an IHCimage identifying tumor and invasive margin tissue, where the IHC may beused to determine the tumor pixels by looking at image colorcharacteristics.

In step 212, the method may include receiving one or more digital imagesassociated with a target pathology specimen and an associated indicationof a presence or an absence of a tumor. The target pathology specimenmay comprise a histology specimen, a cytology specimen, etc. The one ormore digital images may be received into a digital storage device (e.g.,hard drive, network drive, cloud storage, RAM, etc.).

In step 214, the method may include breaking the one or more digitalimages into at least one sub-region to determine if the tumor is presentin the at least one sub-region. Regions may be specified in a variety ofmethods, including creating tiles of the image, edge or contrast,segmentations via color differences, supervised determination by themachine learning model, EdgeBoxes, etc.

In step 216, the method may include applying the machine learning modelto one of the one or more digital images to predict which regions of thedigital image show a tumor tissue or an invasive margin and couldexhibit a biomarker of interest.

In step 218, the method may include, upon determining a sub-regioncontains the tumor tissue or the invasive margin, indicating andflagging a location of at least one tumor region. Detecting the tumortissue and invasive margin regions may be done using a variety ofmethods, including but not restricted to:

-   -   a) Running the trained machine learning model on image        sub-regions to generate the prediction for each image        sub-region.    -   b) Using machine learning visualization tools to create a        detailed heatmap, e.g., by using class activation maps, GradCAM,        etc., and then extracting the relevant regions.

FIG. 2C is a flowchart illustrating an exemplary method of training 220and a method of using a localized biomarker prediction module 230, usingmachine learning, according to an exemplary embodiment of the presentdisclosure. Biomarkers may include genomic results, IHC or outcomeresults. Identification of localized biomarkers from H&E slides mayenable more precision therapy, while reducing cost, turnaround time, andinterobserver interpreter variability. AI-based inference of localizedbiomarkers may provide information on tumor biology and microenvironment(e.g., the interaction of various cell types with a tumor), which mayallow more accurate patient stratification strategies. An AI basedinference of localized H&E based genotyping/molecular/immune markertesting may allow rapid screening of patients that may require a morecomprehensive molecular test, or rapid screening for selecting patientsthat may benefit most in targeted therapies. The below embodiments maybe used to predict any biomarkers that are used in clinical trials suchas flow cytometry, blood assays, etc.

According to one embodiment, the exemplary method of training and usingthe localized biomarker prediction module may include one or more of thefollowing steps. In step 221, the method may include receiving one ormore training digital images associated with a training pathologyspecimen. The training pathology specimen may comprise a histologyspecimen, a cytology specimen, etc. The training digital images may bereceived into a digital storage device (e.g., hard drive, network drive,cloud storage, RAM, etc.).

In step 223, the method may include receiving a plurality of data on alevel of a biomarker present in a tumor and/or an invasive margin regionshown in one or the one or more training digital images. The biomarkerpresence may be indicated with a binary or an ordinal value.

In step 225, the method may include breaking the one or more trainingdigital images into at least one sub-region to determine if a tumor ispresent in the at least one sub-region. Breaking the one or moretraining digital images into sub-regions may be based on sub-regionproperties. Sub-regions may be specified in a variety of methods,including creating tiles of the image, segmentations, based on edges orcontrast, segmentations via color differences, supervised determinationby the machine learning model, etc.

In step 227, the method may include identifying at least one tumorand/or at least one invasive margin region relevant to a biomarker ofinterest. This may be done using an AI-based system or using manualannotations from an expert.

In step 229, the method may include training a machine learning systemto predict an expression level of each biomarker from the at least onetumor and/or the at least one invasive margin region. Expression levelsmay be represented as binary numbers, ordinal numbers, real numbers,etc. This algorithm may be implemented in multiple ways, including butnot limited to:

a) CNN

b) CNN with MIL

c) Recurrent Neural Network

d) Long-short term memory RNN (LSTM)

e) Gated recurrent unit RNN (GRU)

f) Graph convolutional network

g) Support vector machine

h) Random forest

In step 232, the method may include receiving one or more digital imagesassociated with a target pathology specimen. The target pathologyspecimen may comprise a histology specimen, a cytology specimen, etc.The one or more digital images may be received into a digital storagedevice (e.g., hard drive, network drive, cloud storage, RAM, etc.).

In step 234, the method may include receiving a location of a tumor andan invasive margin region. The location may be automatically or manuallyspecified by an expert.

In step 236, the method may include applying a trained machine learningsystem to output a prediction of a biomarker expression level in atleast one region of interest.

In step 238, the method may include outputting a biomarker expressionlevel prediction to an electronic storage device. The method mayadditionally include generating a visual indicator to alert the user(e.g., a pathologist, a histology technician, etc.) to the presence ofthe biomarker.

FIG. 2D is a flowchart illustrating an exemplary method of training andusing the biomarker comparison module, using machine learning, accordingto an exemplary embodiment of the present disclosure. The biomarkercomparison module may take, as input, spatially organized biomarkersignatures or vector embeddings of the spatially organized biomarkersand infer information using AI, e.g., treatment outcome, treatmentresistance, etc. Exemplary methods 240 and 250 may be performed by slideanalysis tool 101 automatically or in response to a request from a user.

According to one embodiment, the exemplary methods 240 and 250 fortraining and using the biomarker comparison module may include one ormore of the following steps. In step 242, the method may includereceiving a spatially structured training input associated with atraining input from a localized biomarker prediction module. Thespatially structured input from the localized biomarker predictionmodule may comprise information about whether the location lies in atumor, an invasive margin, outside the tumor, etc.

In step 244, the method may include receiving a plurality of metadatacorresponding to each spatially structured training input. The metadatamay comprise demographic information, patient history, etc.

In step 246, the method may include training the machine learning systemto predict a treatment outcome or a resistance prediction from alocalized biomarker. Training the machine learning system may comprisean algorithm implemented in multiple ways, including but not limited to:

a. CNN

b. CNN trained with MIL

c. Recurrent neural network

d. Long-short term memory RNN (LSTM)

e. Gated recurrent unity RNN (GRU)

f. Graph convolutional network

g. Support vector machine

h. Random forest

In step 252, the method may include receiving a spatially structuredinput from a localized biomarker prediction module. The input from thelocalized biomarker prediction module may include high-level variablesor vector embeddings. Each spatial structure location may containinformation about whether the location lies in the tumor, invasivemargin, outside the tumor, etc.

In step 254, the method may include receiving a plurality of meta-datacorresponding to the spatially structured input (e.g., demographicinformation, patient history, etc.).

In step 256, the method may include applying the machine learning modelto predict the treatment outcome or a resistance prediction from alocalized biomarker.

In step 258, the method may include outputting a treatment outcome orresistance prediction to an electronic storage device. The method mayalso include generating a visual indicator to alert a user (e.g., apathologist, histology technician, etc.) to the outcome information.

FIG. 3 is an exemplary system trained to detect immune markers within atumor and surrounding invasive margin, according to an exemplaryembodiment of the present disclosure. The exemplary embodiment 300(e.g., steps 302-310) may be performed by slide analysis tool 101automatically or in response to a request from a user.

Exemplary embodiment 300 presents the steps for detecting immune markerswithin the tumor and surrounding invasive margin of the tumor which hasprognostic implications for cancer recurrence, metastasis and treatmentresponse. Embodiment 300 may include one or more of the following steps.In step 302, the method may include an algorithm is fed a digital wholeslide image of a breast tissue, where some of the tissue is cancerous.In step 304, the method may include the salient tissue is detected bythe algorithm. In step 306, the method may include a tumor tissuedetector and invasive margin detector that may filter the image to focuson specific tissue regions that have cancer and neighboring non cancertissue regions. The detection of invasive or neighboring non cancertissue regions can be performed via various morphological or clusteringapproaches etc. In a step 308, the method may include an AI that mayinfer an expression level of each immune marker within the tumor andsurrounding non-tumor region, using the tumor regions. In a step 310,the method may include detecting immune markers within the tumor and mayfurther include detecting immune markers within the surrounding invasivemargin region. Step 310 may be performed partly or entirely usingmachine learning.

FIG. 4 is a flowchart illustrating an exemplary method for training andusing an immune marker localization model, according to an exemplaryembodiment of the present disclosure. Identification of immune markers(e.g., tumor infiltrating lymphocytes (CD3 T-cells, CD8 T-cells, etc.),macrophages (CD68, CD163, etc.) may better characterize a patient'simmune system and help to assess which patients are good candidates forimmunotherapies. High levels of tumor infiltrating lymphocytes withinand in an invasive margin of a tumor may be a good prognostic marker forcancer (e.g., Immunoscore). IHC may be used by pathologists to identifythe expression and localization of these critical markers in tumor andinvasive margin of the tumor tissue. In recent years, next generationsequencing technologies such as RNA-sequencing and flow cytometry havealso been used for immunophenotyping, however the tissue architectureand any information about the spatial relationship between differentcells may be lost in these technologies.

This embodiment comprises applying AI to predict immune markers from H&Estained digital images from various immunophenotyping methods. Thisembodiment may use the tumor/invasive margin region detector to identifytumor and non-surrounding non-tumor regions. The exemplary methods 400and 420 (e.g., steps 402-408 and steps 422-428) may be performed byslide analysis tool 101 automatically or in response to a request from auser.

According to one embodiment, the exemplary method 400 for training theimmune marker localization model may include one or more of thefollowing steps. In step 402, the method may include receiving one ormore digital images of a tissue specimen stained with H&E into a digitalstorage device (e.g., hard drive, network drive, cloud storage, RAM,etc.).

In step 404, the method may include identifying at least one tumorregion and a surrounding tumor tissue in each received image, usingeither an AI-based method or manual specification.

In step 406, the method may include receiving, for each image, anindication of one or more of the immune markers (e.g., CD3, CD8, CD68,etc.). The level of immune marker expression may be identified usingIHC, flow cytometry, RNA sequencing, etc. The level of expression may beon a numeric, ordinal, or binary scale. The indication may be assignedto the entire image or image subregions.

In step 408, the method may include training an immune markerlocalization machine learning model to predict the level of the immunemarker present from the tumor and invasive margin regions of each of thereceived digital images of the pathology specimen.

In step 422, the method may include receiving one or more digital imagesof a selected pathology specimen into a digital storage device (e.g.,hard drive, network drive, cloud storage, RAM, etc.).

In step 424, the method may include identifying tumor image regions thatcorrespond to tumor and surrounding non-tumor tissue in each receivedimage. This step may be performed by an AI-based method (e.g., thetumor/invasive margin region detection model) or manual specification.

In step 426, the method may include applying the machine learning markerlocalization model to at least one received image to output a predictionof an expression level or an immune marker.

In step 428, the method may include outputting a prediction of anexpression level of an immune marker to an electronic storage device.The output may comprise generating a visual indication to alert the user(e.g., a pathologist, histology technician, etc.) of the expressionlevels of each immune marker. The output may additionally recommendtreatments that may be effective for the tumor, given the predictedimmune markers and their predicted expression levels.

FIG. 5 is a flowchart illustrating an exemplary method of training andusing a machine learning model for localization of gene signaturesand/or mutations in pathology specimens. Genetic and molecular testingof cancer tissue may allow for precision treatment of solid tumors viatargeted therapies. Even though the cost of genome sequencing hassubstantially decreased over the years, these tests may still be costly,slow, and require substantial amounts of tissue, tissue that may belimited in clinical studies. Furthermore, the tissue architecture andany information about the spatial relationship between different cellsmay be lost during these tests. This embodiment may infer localizedgenetic and molecular biomarkers (such as the over-expression of aprotein and/or gene product, amplification, mutations of specific genes,etc.) from digital images of pathology specimens. Exemplary methods 500and 520 (e.g., steps 502-508 and steps 522-530) may be performed byslide analysis tool 101 automatically or in response to a request from auser.

According to one embodiment, the exemplary method 500 for training amachine learning model for localization of gene signatures and/ormutations in pathology specimens may include one or more of thefollowing steps. In step 502, the method may include receiving one ormore digital images of a tissue specimen into a digital storage device(e.g., hard drive, network drive, cloud storage, RAM, etc.).

In step 504, the method may include identifying tumor images regionscorresponding to cancerous tissue in each received image, using eitheran AI-based model (e.g., the tumor region detection model) or manualspecification.

In step 506, the method may include receiving, for each image, anindication of the presence of one or more of a gene signature or a genemutation. The presence of the mutations may be identified usingvalidated sequencing methods. The presence of the mutation may bereported as a categorical variable, and its variant allele fraction andcancer cell fraction (e.g., the bioinformatically-inferred percentage ofcancer cells in a sample harboring a given mutation) may be reported ona numeric, ordinal, or binary scale. The indication may be assigned tothe entire image or image sub-regions (e.g., tumor).

In step 508, the method may include training a gene signature and/ormutation biomarker localization machine learning model to predict alevel of a mutation present from each spatial region within the set ofdigital images of the pathology specimens.

In step 522, the method may include receiving one or more digital imagesof a selected tissue specimen into a digital storage device (e.g., harddrive, network drive, cloud storage, RAM, etc.).

In step 524, the method may include identifying tumor image regions thatcorrespond to cancerous tissue for the received images, using either anAI-based method (e.g., the tumor detection model) or manualspecification.

In step 526, the method may include applying the trained gene signatureand/or mutation biomarker localization machine learning model to theimage to output a localization of the gene mutation.

In step 528, the method may include assigning the localized presence ofthe gene mutation to a diagnostic category.

In step 530, the method may include outputting the gene mutation, genemutation expression level, gene mutation location or diagnostic categoryprediction to an electronic storage device. The output may compriseusing a visual indicator to assert the user (e.g., a pathologist,histology technician, etc.) of the expression levels and location ofeach gene mutation.

FIG. 6 is a flowchart illustrating an exemplary method of training andusing a machine learning model for localization of biomarkers in thetumor microenvironment for immunotherapy response prediction.Recognition of the tumor cells by the immune system for destruction mayentail a set of conditions that can be utilized in several biomarkersthat are utilized for assessment of the potential efficacy ofimmunotherapies including antibodies against PD1, PDL1, among others.Some of these biomarkers include the number of somatic mutations (e.g.,tumor mutation burden), IHC for markers including MSI (microsatelliteinstability), PDL1 and PD1, gene expression signatures for the level ofinflammation in the microenvironment, among others. In addition to thequantification of the biomarkers, the location of biomarkers withrespect to the tumor may also provide critical information inunderstanding or predicting a therapeutic response.

The embodiment may be used to identify and localize biomarkers in atumor microenvironment to better understand an immune landscape ofpatients and their likelihood of responding to immunotherapies.According to the embodiment, the exemplary methods 600 and 620 (e.g.,steps 602-610 and steps 622-628) for training a machine learning modelfor localization of biomarkers in the tumor microenvironment forimmunotherapy response prediction may include one or more of thefollowing steps. In step 602, the method may include receiving one ormore digital images of a cancer tissue specimen into a digital storagedevice (e.g., hard drive, network drive, cloud storage, RAM, etc.). Foreach received image, the method may also include receiving an indicationof the presence or absence of the tumor region, e.g., cancerous tissue.

In step 604, the method may include receiving the tissue specimen typeof the cancer tissue specimen.

In step 606, the method may include identifying tumor image regions thatcorrespond to tumor and surrounding non-tumor tissue using either anAI-based method (e.g., the tumor region detection model) or manualspecification.

In step 608, the method may include receiving, for each image, anindication of the sensitivity to a checkpoint inhibitor, tumor mutationburden, MSI inflamed tumor microenvironment or PDL1/PD1 positivity.These presences may be reported in a categorical scale (e.g., presentvs. absent). The indication may be assigned to the entire image or imagesub-regions.

In step 610, the method may include training an immune responsebiomarker localization machine learning model to predict the level ofthe biomarker present from the (tumor and invasive margin) regions ofeach received image.

In step 622, the method may include receiving one or more digital imagesof a selected cancer pathology specimen into a digital storage device(e.g., hard drive, network drive, cloud storage, RAM, etc.).

In step 624, the method may include receiving a tissue specimen type ofthe selected cancer pathology specimen.

In step 626, the method may include identifying, for the selected image,tumor image regions that correspond to tumor and invasive margin of thetissue using either an AI-based method (e.g., the tumor region detectionmodel) or manual.

In step 628, the method may include applying the immune responsebiomarker localization machine learning model to at least one receivedimage to predict the localization or expression level of a biomarker inthe tumor and invasive margin. The machine learning model may includethe following steps:

-   -   a. Assigning the presence of a biomarker to a diagnostic        category    -   b. Outputting the predicted localization or expression levels of        the biomarker to an electronic storage device    -   c. Generating a visual indicator to alert the user (e.g., a        pathologist, histology technician, etc.) of the predicted        expression levels of the biomarkers.

FIG. 7 is a flowchart illustrating an exemplary method of using machinelearning to predict antineoplastic resistance, according to an exemplaryembodiment of the present disclosure. Antineoplastic resistance mayoccur when cancer cells resist and survive despite anti-cancertreatments. This ability may evolve in cancers during the course oftreatment, and predicting which therapies the cancer will have the mostdifficulty acquiring resistance to may improve patient treatment andsurvival. Some cancers may develop resistance to multiple drugs over thecourse of treatment. The present embodiment may predict the probabilityof antineoplastic resistance by examining the environment within andexternal to a tumor.

According to one embodiment, the exemplary method 700 for training anantineoplastic resistance prediction system using AI may include one ormore of the following steps. In step 702, the method may includereceiving one or more digital images of a cancer tissue specimen (e.g.,stained with H&E) into a digital storage device (e.g., hard drive,network drive, cloud storage, RAM, etc.).

In step 704, the method may include receiving, for each of the digitalimages, a corresponding tissue specimen type.

In step 706, the method may include receiving, for each of the digitalimages, data regarding a treatment given to a patient associated withthe tissue specimen and the outcome (e.g., whether antineoplasticresistance occurred). Exemplary outcomes can be at one time point ormultiple time points.

In step 708, the method may include identifying tumor image regions ineach of the digital images that correspond to tumor and surroundingnon-tumor tissue, using either an AI-based method (e.g., the tumorregion detection model) or manual specification.

In step 710, the method may include training a resistance predictionmachine learning model, e.g., a deep neural network, to predict anoutcome for the treatment (e.g., if antineoplastic resistancedeveloped). This classification may be done using a multi-class ormulti-label approach, with treatments that were not given handled asmissing values.

Method 720 may be implemented when using the trained system inproduction. In step 722, the method may include receiving one or moredigital images of a selected cancer pathology specimen into a digitalstorage device (e.g., hard drive, network drive, cloud storage, RAM,etc.).

In step 724, the method may include receiving a tissue specimen type ofthe selected cancer pathology specimen.

In step 726, the method may include identifying, for the selected image,at least one tumor region that corresponds to tumor and invasive marginof the tissue, using either an AI-based method (e.g., the tumor regiondetection model) or manual specification.

In step 728, the method may include applying the trained resistanceprediction machine learning model to at least one received image of theselected cancer pathology specimen to predict a treatment responseoutcome for one or more treatment types. The prediction may includewhether any antineoplastic resistance will occur to each treatment type.

In step 730, the method may include outputting the treatment outcome andantineoplastic resistance prediction to an electronic storage device.The output may be in the form of a visual indicator to alert the user(e.g., a pathologist, histology technician, etc.) of the treatments thatare predicted to be ineffective due to antineoplastic resistancedeveloping. The output may further include recommending treatments basedon the predictions, and outputting the treatments to the user or to anelectronic storage device.

As shown in FIG. 8 , device 800 may include a central processing unit(CPU) 820. CPU 820 may be any type of processor device including, forexample, any type of special purpose or a general-purpose microprocessordevice. As will be appreciated by persons skilled in the relevant art,CPU 820 also may be a single processor in a multi-core/multiprocessorsystem, such system operating alone, or in a cluster of computingdevices operating in a cluster or server farm. CPU 820 may be connectedto a data communication infrastructure 810, for example, a bus, messagequeue, network, or multi-core message-passing scheme.

Device 800 also may include a main memory 840, for example, randomaccess memory (RAM), and also may include a secondary memory 830.Secondary memory 830, e.g., a read-only memory (ROM), may be, forexample, a hard disk drive or a removable storage drive. Such aremovable storage drive may comprise, for example, a floppy disk drive,a magnetic tape drive, an optical disk drive, a flash memory, or thelike. The removable storage drive in this example reads from and/orwrites to a removable storage unit in a well-known manner. The removablestorage unit may comprise a floppy disk, magnetic tape, optical disk,etc., which is read by and written to by the removable storage drive. Aswill be appreciated by persons skilled in the relevant art, such aremovable storage unit generally includes a computer usable storagemedium having stored therein computer software and/or data.

In alternative implementations, secondary memory 830 may include othersimilar means for allowing computer programs or other instructions to beloaded into device 800. Examples of such means may include a programcartridge and cartridge interface (such as that found in video gamedevices), a removable memory chip (such as an EPROM, or PROM) andassociated socket, and other removable storage units and interfaces,which allow software and data to be transferred from a removable storageunit to device 800.

Device 800 also may include a communications interface (“COM”) 860.Communications interface 860 allows software and data to be transferredbetween device 800 and external devices. Communications interface 860may include a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, or the like. Software anddata transferred via communications interface 860 may be in the form ofsignals, which may be electronic, electromagnetic, optical, or othersignals capable of being received by communications interface 860. Thesesignals may be provided to communications interface 860 via acommunications path of device 800, which may be implemented using, forexample, wire or cable, fiber optics, a phone line, a cellular phonelink, an RF link or other communications channels.

The hardware elements, operating systems and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. Device 800 alsomay include input and output ports 850 to connect with input and outputdevices such as keyboards, mice, touchscreens, monitors, displays, etc.Of course, the various server functions may be implemented in adistributed fashion on a number of similar platforms, to distribute theprocessing load. Alternatively, the servers may be implemented byappropriate programming of one computer hardware platform.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules can be implemented in software,hardware, or a combination of software and hardware.

The tools, modules, and functions described above may be performed byone or more processors. “Storage” type media may include any or all ofthe tangible memory of the computers, processors or the like, orassociated modules thereof, such as various semiconductor memories, tapedrives, disk drives and the like, which may provide non-transitorystorage at any time for software programming.

Software may be communicated through the Internet, a cloud serviceprovider, or other telecommunication networks. For example,communications may enable loading software from one computer orprocessor into another. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

The foregoing general description is exemplary and explanatory only, andnot restrictive of the disclosure. Other embodiments of the inventionwill be apparent to those skilled in the art from consideration of thespecification and practice of the invention disclosed herein. It isintended that the specification and examples be considered as exemplaryonly.

1-20. (canceled)
 21. A computer-implemented method, comprising: receiving one or more digital images of a pathology specimen from a patient, the one or more digital images comprising data about one or more biomarkers and a surrounding invasive margin region; and generating, using a machine learning model on the one or more digital images, at least one inference of a presence of the one or more biomarkers in the pathology specimen and the surrounding invasive margin region, wherein the machine learning model is configured to compare a spatial relationship of each of the one or more biomarkers and the surrounding invasive margin region by receiving at least one spatially structured input from a localized biomarker prediction module and predict a treatment outcome and/or at least one treatment recommendation for the patient.
 22. The computer-implemented method of claim 21, wherein generating the at least one inference of the presence of the one or more biomarkers comprises receiving metadata corresponding to the at least one spatially structured input.
 23. The computer-implemented method of claim 21, wherein the at least one spatially structured input from the localized biomarker prediction module comprises a plurality of vector embeddings.
 24. The computer-implemented method of claim 21, wherein the at least one spatially structured input comprises a spatial structure location comprising one or more location lies in a tumor and/or the surrounding invasive margin region.
 25. The computer-implemented method of claim 21, further comprising identifying tumor tissue and the surrounding invasive margin region for each of the one or more digital images by the machine learning model.
 26. The computer-implemented method of claim 25, further comprising: receiving one or more training digital images associated with a training pathology specimen and an associated indication of a presence of a tumor region; and dividing the one or more training digital images into at least one sub-region to determine if the tumor tissue is present in the at least one sub-region.
 27. The computer-implemented method of claim 25, further comprising: receiving one or more digital images associated with a target pathology specimen and an associated indication of a presence of a tumor region; dividing the one or more digital images into at least one sub-region to analyze to determine if the tumor tissue is present in the at least one sub-region; applying the machine learning model to one of the one or more digital images to predict which regions of each digital image of the one or more digital images show a tumor tissue; and indicating and flagging a location of at least one tumor region.
 28. The computer-implemented method of claim 24, further comprising: receiving one or more training digital images associated with a training pathology specimen and an associated indication of an absence of a tumor region; and dividing the one or more training digital images into at least one sub-region to determine if a tumor tissue is absent in the at least one sub-region.
 29. The computer-implemented method of claim 21,wherein generating the at least one inference of the presence of the one or more biomarkers further comprises: receiving one or more training digital images of the pathology specimen; receiving a plurality of data on a level of the one or more biomarkers present in a tumor and/or an invasive margin region shown in one of the one or more training digital images; dividing one of the one or more training digital images into at least one sub-region to determine at least one property for the at least one sub-region; and identifying at least one tumor and/or at least one invasive margin region relevant to a biomarker of interest.
 30. The computer-implemented method of claim 21, wherein generating the at least one inference comprises determining at least one region of interest in a tumor tissue, and applying the machine learning model to determine a prediction of a biomarker expression level in the at least one region of interest.
 31. The computer-implemented method of claim 21, wherein generating the at least one inference comprises applying the machine learning model to predict the treatment outcome from the localized biomarker.
 32. A system, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions to perform operations comprising: receiving one or more digital images of a pathology specimen from a patient, the one or more digital images comprising data about one or more biomarkers and a surrounding invasive margin region; and generating, using a machine learning model on the one or more digital images, at least one inference of a presence of the one or more biomarkers in the pathology specimen and the surrounding invasive margin region, wherein the machine learning model is configured to compare a spatial relationship of each of the one or more biomarkers and the surrounding invasive margin region by receiving at least one spatially structured input from a localized biomarker prediction module and predict a treatment outcome and/or at least one treatment recommendation for the patient.
 33. The system of claim 32, wherein the pathology specimen comprises a cytology specimen.
 34. The system of claim 32, wherein the at least one spatially structured input comprises a spatial structure location comprising one or more location lies in a tumor and/or the surrounding invasive margin region.
 35. The system of claim 32, wherein the data associated with the one or more biomarkers is identified from at least one of genetic testing, flow cytometry, and immunohistochemistry.
 36. The system of claim 32, wherein the operations further comprise identifying a tumor tissue and the surrounding invasive margin region using the machine learning model by: receiving one or more training digital images associated with a training pathology specimen and an associated indication of a presence or an absence of a tumor region; and dividing the one or more training digital images into at least one sub-region to determine if the tumor tissue is present in the at least one sub-region.
 37. The system of claim 36, wherein the operations further comprise applying the machine learning model to one of the one or more digital images to predict which regions show a tumor tissue or an invasive margin.
 38. The system of claim 37, wherein the operations further comprise indicating and flagging a location of at least one tumor regions.
 39. The system of claim 32, wherein generating the at least one inference of the presence of the one or more biomarkers further comprises: receiving one or more training digital images of the pathology specimen; receiving a plurality of data on a level of the one or more biomarkers present in a tumor and/or an invasive margin region shown in one of the one or more training digital images; dividing one of the one or more training digital images into at least one sub-region to determine at least one property for the at least one sub-region; and identifying at least one tumor and/or at least one invasive margin region relevant to a biomarker of interest.
 40. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method comprising: receiving one or more digital images of a pathology specimen from a patient, the one or more digital images comprising data about one or more biomarkers and a surrounding invasive margin region; and generating, using a machine learning model on the one or more digital images, at least one inference of a presence of the one or more biomarkers in the pathology specimen and the surrounding invasive margin region, wherein the machine learning model is configured to compare a spatial relationship of each of the one or more biomarkers and the surrounding invasive margin region by receiving at least one spatially structured input from a localized biomarker prediction module and predict a treatment outcome and/or at least one treatment recommendation for the patient. 